Bridging the Gaps between Many-task Computing and Supercomputers

Many Task Computing, an emerging programming paradigm on supercomputers, embraces many applications in such domains as biology, economics, and statistics, as well as data intensive computations and uncertainty quantification. Its high inter-task parallelism and intense data processing features place new challenges on the existing hardware-software stack on supercomputers. Those new challenges include resource provisioning, job scheduling, load balancing, data management, and resiliency. In this paper, we identify Many-Task Computing middleware gaps between the applications and supercomputer’s hardware-software stack by examining their characteristics. Based on this analysis, we propose AME, an Anyscale MTC Engine with a special focus on scalability. We describe the AME framework and present performance results for both synthetic benchmarks and real applications. Our results show that AME is a highly scalable MTC engine on petascale machines, and a strong candidate for exascale machines and beyond.